Skip to content

Ship reusable analyze-action-pr skill under .claude/skills/#807

Open
potiuk wants to merge 3 commits intoapache:mainfrom
potiuk:add-analyze-action-pr-skill
Open

Ship reusable analyze-action-pr skill under .claude/skills/#807
potiuk wants to merge 3 commits intoapache:mainfrom
potiuk:add-analyze-action-pr-skill

Conversation

@potiuk
Copy link
Copy Markdown
Member

@potiuk potiuk commented May 6, 2026

Summary

How to use

In any Claude Code session opened in this repo, ask: "analyze PR 802" (or any PR number). The skill is auto-loaded by Claude based on its description; the trigger phrasing is also documented in the skill body.

Test plan

  • git check-ignore confirms .claude/settings.local.json is still ignored under the new rules.
  • git status from the worktree root shows only the SKILL.md addition and the .gitignore / AGENTS.md diffs (no personal settings file leaks).
  • prek run --all-files is clean (license headers, sort, etc.).
  • CI Pre-commit Checks job passes on the latest commit.
  • Manual: open a fresh Claude Code session in this repo, ask "analyze PR 803" (a clean PR), and confirm the skill triggers and produces the expected workflow output.

Generated-by Claude Code.

Add a repo-scoped Claude Code skill at .claude/skills/analyze-action-pr/
that triages allowlist PRs end-to-end: extract action refs via
verify-action-build --from-pr, classify each finding (pipe-to-shell,
unverified-download, nested-action-issue, verify-script-bug, all-clean),
look up what verification material upstream already publishes, and
draft the right next moves — recommend approval, open an upstream issue
+ ping the PR author, or fix verify-action-build itself with a
regression test.

Includes an "Improve this skill" step so future runs that hit a new
shape (false positive, novel verification mechanism, asset-naming
convention) leave the skill better than they found it: the SKILL.md
grows by a row in the case table or a line in the precedents table,
not by re-deriving the analysis.

The skill encodes shapes recently triaged: PR apache#795 / apache#798 (TS-generic
regex hole), PR apache#802 (carabiner-dev nested install actions), PR apache#803 /
apache#804 (multi-action extraction), PR apache#806 (jbangdev pipe-to-shell).

Replace the broad .claude/ rule in .gitignore with specific entries
for personal state (settings*.json, worktrees/, debug/) so .claude/
itself can host shared assets without leaking per-user files.
@potiuk potiuk requested a review from dfoulks1 as a code owner May 6, 2026 07:25
@potiuk potiuk requested review from dave2wave and raboof May 6, 2026 07:26
Pre-commit on PR apache#807 failed because .claude/skills/analyze-action-pr/
SKILL.md was missing the Apache license header.  The insert-license
hook auto-fixes locally but aborts CI because there's no follow-up
commit to land the change.  Add the header so CI passes.

Document the local pre-commit workflow in AGENTS.md so future agent
runs install prek and run ``prek run --all-files`` before pushing,
catching the same class of bounce locally instead of in CI.
@potiuk
Copy link
Copy Markdown
Member Author

potiuk commented May 6, 2026

cc: @dfoulks1 @raboof @dave2wave @ppkarwasz - > this skill is somethign that any of you could be using to do what I have been doing for a while (I did it mostly to gather material).

I.e.:

a) open your agent
b) run "analyze #NNN` -> NNN is failing PR

All the rest should happen automatically. The agent should propose you what to do - whether to propose an issue in the upstream repo or what to advice to the user who is submitting the action, possibly even open a PR fixing the issue in the upstream action.

This is ultimately the way how we can make what I did here highly scalable and done by any volunteer who is able to simply review what action proposes and to make decisions what to do next.

This SKILL will never post anything on it's own, it will always ask the one who runs it to confirm before anything is done.

If you are afraid about security and credentials -> https://cwiki.apache.org/confluence/display/ASFP/RFC-AI-0002+Secure+Agents+setup is the RFC that describes the setup and https://github.com/apache/airflow-steward/tree/main/.claude/skills/setup-isolated-setup-update performs the secure setup for you - so you have to confirm all actions that require your token or credentials to get out of your sandbox - the steward - to be renamed performs global setup for your account so you can use steward to set it up so that all your claude sessions are isolated by default.

@potiuk
Copy link
Copy Markdown
Member Author

potiuk commented May 6, 2026

@dfoulks1 @raboof @dave2wave @ppkarwasz -> I hope that will help us to distribute the load and review / approve / reject / fix workflow here.

Literally anyone can do what I've been doing for the last few weeks - you do not need special knowledge on what to ask - just "analyze #NNN` when you see a failing PR - in vast majority of cases the agent should make a good proposal what to do - all what you need to do is to review the proposal and confirm.

Case E previously read as "the script has a bug" — too narrow.  Real
gaps over the life of verify-action-build include whole-action-type
support (Deno, Dart), new build flows (npm run start, multi-step
Docker), new verification heuristics (sibling sha256sum, JSON data
fetches, *Json helpers), false positives on legitimate inputs
(multi-stage FROM, vendored node_modules), and extraction-shape gaps
(hashes under an existing actions.yml key).

Reword the case E row to cover all of these, and add a sub-table
under step 5/E that lists 15 historical extensions with the PR or
commit that closed each.  The table is the template a future agent
reaches for when "the verdict is wrong but not for a security
reason" — instead of waving the finding off, they can match it to
the closest precedent and propose a similar extension.

Also tighten the steps: name each module under
utils/verify_action_build/ and what it owns (pr_extraction, security,
verification, docker_build, action_ref/release_lookup/github_client),
so picking the right place for the fix isn't a treasure hunt.  Add
an explicit "run prek before push" step pointing at AGENTS.md.
@potiuk
Copy link
Copy Markdown
Member Author

potiuk commented May 6, 2026

All the actions that I've done now in #805 was result of running :

analyse #805

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant